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ABSTRACT 

Local education agency evaluations vere systematized 
to be combined to provide information about the outcomes of 
occupational programs in the state. This vas achieved by developing a 
manual of evaluation procedures and tools for use by teachers of 
occupation programs. Pre--test and post-test data from 397 students in 
1971 and 287 students in 1972 provided information on many variables, 
including ability, disadvantaged status, knowledge of content^ 
attitudes toveird work, job, and school, and self-evaluation in the 
work. Data from local agencies, when systematized, can be combined to 
provide more comprehensive descriptive information about program 
outcomes. (Author) 
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Systematizing Local Education Agency Evaluations as a Basis for 
Statewide Evaluation of Occupation Courses 

Mary Bach Kievit''- 

Home Economics Occupations Courses have become more firmly estab- 
lished in New Jersey secondary schools since their inception in 1965. Educators 
have become increasingly interested in striving to develop and to improve eval- 
uative processes as an integral part of these courses. The objective of this 
project was to initiate and to continue an effort toward systematizing self- 
evaluations, within local agencies, of funded home economics occupi. lon courses 
in the state. One purpose in systematizing local agency evaluations was to make 
it possible to combine the results from each program, thus providing more com- 
prehensive information about programs throughout the state. 

Overview of Development and Procedures 

This evaluation of home economics related occupation courses was 
initiated in April, 1970. The four phases of the project were implemented 
from April, 1970 through May, 1973. 

Phase I included the following activities: 

1. reviewing the literature related to evaluating educational programs 
generally, and specifically home economics programs. 

2. planning and implementing three conferences with teachers of home 
economics occupation courses, to develop criteria for evaluation, 
to share information about evaluative procedures, and techniques, 
and to identify problems encountered in evaluation of local programs. 

Phase II was planned on the basis of information and insight acquired during 
Phase L During Phase II, a framework for systematizing evaluations in each local 
education agency was developed. This framework was specified and detailed in the 
form of a manual for teachers entitled. Self -E valuation : Home Economics Occupa- 
tion Courses and Co-Operative Education Programs. The manual had two sections: 
Part I, "Spotlight on Student Achievement, " and Part II, "Program Facilitators. " 

Survey instruments, attitude, and content measures pertinent to these two 
broad areas were included in each section. Scoring information and report forms 
were also included. 

A preliminary draft of the mmiual was critiqued by teachers of the courses 
and programs to be evaluated. A revised draft was duplicated and distributed to 
45 teachers for preliminary testing between January and May, 1971. Twenty-five 
teachers returned pre-tesl and post-test data for 397 students enrolled in 27 courses. 

''"Professor, Department of Vocational -Technical Education,' Rutgers, The State 
University, New Brunswick, N. J. 08903 



The preliminary testing provided data with which; 

to check the reliability of some of the measuring 
devices for New Jersey students 

to establish a base line for examining data obtained 
in subsequent years 

to revise the procedures set up in the evaluation 
manual 

Phase III was implemented between August, 1971 and August, 1972. 
•Revised manuals and test materials for each student enrolled in occupation 
courses were distributed to teachers and explained at meetings. A number 
of teachers not attending the meetings received the materials by mail. Teachers 
pre-tested students in the first six weeks of the course and administered the 
post-tests during April. Teachers (22) of 23 courses and programs returned 
usable pre-test and post-test materials for 287 students.. This number of 
programs constituted 37 percent of the 62 occupation programs funded during 
fiscal year 1972, in New Jersey. 

Phase IV involved the analysis of the data and preparation of this report. 
Throughout this analysis, mean score data pertaining to student achievement 
were analyzed with occupational area, ability of students, and disadvantaged 
status treated as independent variables. Post-test mean scores obtained in 
1971 during Phase IX, were incorporated into tables to serve as a base line 
for interpreting pre-test and post-test scores obtained in 1971-72. 

Measurement 

Questionnaires and measuring devices from which the findings were 
derived are described in detail in the body of this report. Generally, the level 
of reliability was acceptable. Split half reliability correlation coefficients 
ranged between . 6 and . 7 on the measures of course content, with a wider 
range on attitude measures. There was evidence of reliability levels being 
lower for below average ability students. Concern existed that the vocab- 
ulary level on some questions may have exceeded the reading skills of below 
average students. It is possible that reduced reliability may be linked to the 
level of reading difficulty in some items. These are limitations to be acknowl- 
eged in interpreting the findings derived from student responses. 
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Findings 

Program Characteristics, Teacher Characteristics, and Teacher Evaluations 

Length of program. In 1972, there w'as evidence of a slight shift to 
increasing tlie length of the programs and to increasing class time, as con- 
trasted to 1971. 

Work. In 1971 and 1972, teachers reported simulated work experience in 
approximately 60 percent of the programs. Tho remaining 40 percent provided 
real work experience. 

Employment Surveys. An increased proportion, almost 20 percent of 
teachers in 1972 reported having completed surveys of the local employment 
situation, as compared with teacher reports for 1971. 

Advisory Councils . Advisory Councils seem to have a relatively minor 
influence on programs. In 1971, 13 out of 25 teachers reported a total of 17 
meetings; and in 1972, 7 teachers out of 22 reported 9 meetings. 

Enrollment . Average enrollment per class was 16.8 in 1971 and 13.7 in 

1972. 

Age and Education. The majority of the teachers were over 40 years of 
age and have Bachelors degrees. 

Attitudes . The large majority of teachers had strong positive attitudes 
for these occupational courses and felt quite competent to teach these courses. 

A smaller majority expressed satisfaction witli the characteristics of 
students enrolled in the courses. 

Inadequacies . Public relations, evaluation of students, facilities and 
resources were most frequently cited as inadequacies in the program. 

Student Characteristics and Performance 

Ability . There was an increase of about 13 percent from 49 percent in 
1971 to 63. 7 percent in 1972 in the proportion of students categorized as having 
average ability, with decreases in the above average and below average categories 
between 1971 and 1972. 

Low Income, Minority, Disadvantaged . Students from low income families 
increased proportionally by about 22 percent (43 percent, 1971; 65 percent, 1972); 
from minority groups, by 8 percent (16 percent, 1971; 24 percent, 1972); and 



being disadvantaged, by 20 percent, from 41 percent in 1971 to 61 percent in 1972. 



Home Economics , About 60 percent of the students were reported to have 
had previous experience in home economics. 

Attendance. Students enrolled in the majority of courses attended on the 
averse 2 l/2 days more as a result of being absent less frequently than the pre- 
ceeding year. Teachers reported a loss of approximately 9 percent of the 
students pre-tested in 1971-72. 

Performance on Course Content. Mean score analysis for 1971-72 by 
occupational and ability levels showed students achieving scores that clustered 
around the theoretical means on both pre-tests and post-tests. Somewhat con* 
trary to expectations, only students enrolled in clothing related occupation courses 
and cooperative education programs showed some gain between pre-test and post- 
test scores. 

A comparison of mean scores for advantaged and disadvantaged categories 
of students found no difference for students enrolled in food service courses, a 
5 to 9 point difference for students enrolled in clothing related, child care, and 
cooperative education programs in the direction of higher mean scores for those 
categorized advantaged. 

Attitude Towards Work. The students in each occupational area, ability 
level category, and whether advantaged or disadvantaged have positive attitudes 
toward work. Cooperative education students had slightly more positive attitudes 
than students enrolled in the three other areas, with students in child care ranking 
last. Disadvantaged students did not differ sharply from advantaged students in 
terms of mean scale scores, except in child care on post-test results, and on 
pre-test scores for cooperative education students. In general, responses on the 
scale did not show change consistently between pre-test and post-test. 

Self-Concepts of Ability . Students enrolled in home economics related 
occupation courses tend generally to have favorable self-concepts of ability. The 
mean scores suggest that these self-concepts may be fairly objective, if one 
accepts the assumption that teacher ratings of ability are reasonably reliable 
and valid ratings. Furthermore, during the school year a statistically signi- 
ficant number of students apparently developed more favorable opinions of their 
work and their ability to earn above average grades. 

Attitudes Toward School. Student^s attitudes toward school were positive 
as evidenced by mean scores equal to or exceeding the tlieoretical mean of the 
scale. Variations between occupational areas, advantaged -disadvantaged 
categories, and pre-tests and post-tests were relatively small and in no con- 
sistent direction. Item analysis found a statistically significant (. 05<p<. 10) 
shift in responses on the pre-test and post-test in the direction of a larger pro- 
portion of students reporting their problems to be average, rather than at either 
extreme. 
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Attitudes Toward Home Economics , Students enrolled in home economics 
related occupation courses hold positive attitudes towards home economics. 
Variations between means by occupational areas suggest no marked differences 
in students on this variable. The findings also indicate that this positive attitude 
was not modified to any extent by the instruction in the occupation courses. 

Attitude Towards Job . Mean scores indicate that, in general, students 
tend to hold positive attitudes towards the type of job for which the course prepares 
them. Significant differences did not appear for the various occupational areas, 
nor do students categorized as advantaged differ from those categorized as dis- 
advantaged. The occupation courses do apparently help some students to 
crystallize preferences for some types of work and dislike for others as in- 
dicated by an approximate ten percent shift in responses on two specific items. 
The shifts were a decrease in commitment to the length of time students reported 
planning to work in the job and in liking the type of work. 

Interest in Occupational Training. Students indicate an average interest 
in occupational training. Although not a strong relationship, some of the findings 
suggest an increase in interest for occupational training of the average and below 
average ability categories. There was no relationship with occupational area. 

Tliis relationship finds some corroboration in the results for advantaged 
and disadvantaged categories of students. First, in that variations between 
categories were minimal in general, and secondly that the mean scores of the 
disadvantaged tended to increase from pre-test to post-test in the same degree 
or slightly more so than advantaged students. 

Some ambiguities are apparent in the items, and thus are open to several 
interpretations. Specifically, a decrease in the number agreeing, at the time of 
the post-test, that they need to prepare for a job after high school may indicate 
that they believe the training they are acquiring is good and obviates further 
need. On the other side one could assess this as an inadequacy in the instruction 
which did not sensitize and motivate students to set higher goals which might 
require more training. 

Similarly, the decrease, at the time of the post-test, of those agreeing 
that they need to learn **how to go about getting a job** suggests that for some the 
preparation has given them a sense of adequacy in this sphere. The same applies 
to the significant shift in responses regarding behavior on job interviews. Courses 
apparently helped move some students in the direction of believing some prefer- 
ences and aptitudes for some types of work were emerging as evidenced by the 
shift from ''agreeing^* that they didnU know the job for which they were suited to 
being *' uncertain. 

Attitudes Towards Working Witli Others. With few exceptions mean scores 
of ability categories by occupationai area were at or above the theoretical mean 



of the scale. Post-test scores as well as pre-test scores clustered close around 
the mean, and indicated no or minimal change between pre-test and post-test. 



Advantaged and disadvantaged students did not differ from each other to 
any extent of practical significance. 

Self-Evaluation in the World of Work . Students in each occupational area 
showed gains between pre-test and post-test scores with several exceptions for 
the below average abiUty categories, one in food service where a loss occurred, 
and in co-operative education where the change was so small (. 8)that stability 
seems indicated. Mean scores did cluster around the theoretical mean of the 
scale, and frequently exceeded it. Steady progression of means for some 
occupational areas suggest a positive linear relationship with ability, however, 
this was not a consistent pattern. The lack of consistency may be in part, a 
function of the lower reliability of the scale for below avers^e students. 

A comparison of advantaged and disadvantaged students by occupational 
area indicated that in three* of the four cases, mean scores of tlie disadvantaged 
were somewhat lower than the mean scores for advantaged students. However, 
disadvantaged students i.e. , clothing related and co-operative education, made 
larger gains between pre-test and post-test. 

On the basis of items on this scale in which significant shifts occurred 
between pre-test and post-test, one could infer (alternative interpretations are 
possible) that as the course neared completion, larger proportions of students 
felt more ready to get a job in the particular area; increased in feelings of 
confidence relative to work; were able to admit mistakes; were less self-con- 
scious; had more faith in their abilities and were less easily discouraged; 
were able to give preference to group needs over individual wishes; and res- 
pected persons who believed differently than they. 

In conclusion, items on this scale tapped more definitively the possible 
affective gains from course instruction than other measures included in the 
student questionnaire. 

Self -Confidence . Mean scores of students enrolled in the four occupa- 
tional areas were not markedly different on general self-confidence either at 
the time of the pre-test or post-test. The picture which emerged is a group 
of students with a modest and probably realistic self -concept of confidence 
which was not altered to any measurable degree during participation in the 



*In the fourth case, the mean of disadvantaged students exceeded the 
mean of advantaged students. 
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course. Furthermore, only among students in child care was there much 
discernable difference between those categorized, advantaged and disadvan- 
taged* 

Interest in Earning Money , Students enrolled in home economics 
occupation courses were about average in their interest in earning money. 
The level of interest was not measurably influenced during course instruc- 
tion. 

Expectations from course . On the basis of mean scores most students 
enrolled in home economics occupation courses had favorable expectations for 
the course as indicated by mean scores exceeding the theoretical mean of the 
scale. Although some variations did occur between pre-test and post-test means, 
these were small for all ability categories in each occupational area and for 
advantaged and disadvantaged categories. 

Teacher Ratings of Student Behaviors, Teachers rated students on ten 
attributes as observed by the teacher. These were: appearance on the job; 
co-operation with co-workers; attitudes toward regulations, acceptance of 
supervision, management, dependability, adaptability, attitudes towards public, 
pride on job, and suitability. 

Most categories of students had mean teacher ratings of acceptable, or 
above, on appearance on the job. Patterns of variation suggest a relationohip 
between ability categories and teacher ratings of appearance. A positive 
relationship is suggested between being categorized as advantaged and receiving 
higher teacher mean ratings, except for students in co-operative education 
where the relationship was a negative one. 

Comparisons of pre-test and post-test ratings suggest that the instruc- 
tion in home economic occupation courses generally had minor, if any influence 
on appearance. 

Relative to cooper ativeness , on the basis of mean score analysis, 
students enrolled in home economics occupation courses were rated acceptable 
or above, with several exceptions being rated marginal to acceptable. Mean 
ratings for students in food service courses tended to be somewhat lower than 
for students in other areas. Consistent and marked differences between ad- 
vantaged and disadvantaged categories of students were not evident. Those 
disadvantaged had a lower mean rating in the areas of food service and cloth- 
ing related occupations but had a higher mean rating in co-operative education. 
Slight gains were made betu'een pre-test and post-test mean ratings, except 
in the cases of several ability categories. 

With reference to attitudes towards ^regulations , food service student 
categories tended to be rated the lowest; co-operative education students, the 
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highest, with clothing and child care areas in between. With the exception of 
food service, all categories of students were rated marginal to acceptable, 
acceptable, or good. 

This may in part, be related to the greater importance of regulations 
regai'ding tlie handling of food to maintain hj^gienic conditions. Since behavior 
in these areas has, in large part, become habitual over the years, alteration 
of conduct is likely to be more difficult. 

Acceptance of supervision was defined operationally as the ease of 
accepting criticism. Students enrolled in home economics occupations were 
rated as generally receptive to criticism. Ratings were lower for those in 
food service than in the other tliree areas. Slight gains between pre-test and 
post-test ratings were reported for two of three ability categories in food 
service, all three abilit}' categories in clothing related occupations, and two 
of three ability categories in co-operative education. Relative to advantaged 
and disadvantaged, differences were not consistent nor large. While disad- 
vantaged had lower mean ratings for food service, this category had the 
higher mean rating for co-operative education. Some evidence suggested a 
positive relationship between ability and higher ratings but this was not con- 
sistently the case. 

Relative to management, students in food service were rated the lowest, 
followed by child care, clothing related, and co-operative education. For the 
last three occupational areas mentioned above, mean ratings exceeded the the- 
oretical mean, except for low ability students in child care. Thus, students 
were rated as acceptable to good. Variations between advantaged and disadvan- 
taged showed lower means for the disadvantaged in food service and clothing 
related occupation courses, and a higher mean in co-operative education. 

^ Dependability was operationally defined as punctuality^ loyalty and 
honesty. The lowest mean dependability ratings were obtained by food service 
students, and the highest by those in co-operative education. Gains between pre- 
cast and post-test though sometimes slight were reported for each ability category 
and each occupational area. Advantaged students had higher mean ratings tlian 
disadvantaged in food service and clothing related occupations. There was no 
difference between these categories of co-operative education students. Gen- 
erally, then, categories of students were rated between acceptable and good on 
dependability and showed gains between the pre-test and post-test. 

With reference to adaptability and initiative, all categories of students 
received acceptable and above ratings, with the single exception of the below 
average ability category in food service. Generally, disadvantaged students 
received lower ratings than advantaged except those in co-operative education. 

Attitudes of students toward the public were rated to be acceptable, good 



ERLC 



-8- 



in one case, and very good by the time of the post-test in three of the four 
occupational areas. Jxi food service only those of average ability were rated 
acceptable, the remaining two categories were less than acceptable. With 
the exception of food servic-s ratings were positively related with ability 
level. With the exception of co-operative education, ratings were positively 
related to being categorized as advantaged. Generally, low to modest gains 
between pre-test and post-test were made in ratings of each category. 

Mean ratings on pride in job at the time of the post-test were accept- 
able to good with the exception of Ability categories 1 and 3 in food service. 
Mean ratings tended to be positively related to ability level, although this did 
not hold for food service. MeaXi ratings in three of the four occupational areas 
tended to be positively related to advantaged-disadvantaged status, however, 
except in food service, disadvantaged students were rated acceptable at the 
time of the post-test. Generally, slight gains between pre-test and post-test 
scores were reported for most categories. 

Relative to suitability, with few exceptions, mean ratings of students 
enrolled in home economics occupation courses indicated an acceptable level. 
Trends of progressions of means showed mean ratings tended to increase as 
ability increased and students were categorized as advantaged. Mean ratings 
for co-operative education students were the exception to this. Slight gains 
between pre-test and post-test were reported for most categories. 

Follow-Up of 1970 and 1971 Graduates. Follow-up data were obtained 
for approximately 72 percent (N~201) of the students surveyed by participating 
teachers in 1971. Approximately 88 percent (N=114) were returned from the 
28 surveys sent out by participating teachers in 1972. Findings indicated that 
between 30 and 45 percent of students did not take jobs upon graduation. Of 
those employed, the largest number (i.e. between 50 to 70 percent) obtained 
jobs in areas for which the course had prepared them. Of students employed 
in jobs for which the course prepared them in excess of 90 percent for 1971 
and 1972 reported that they like their work; this compared with approximately 
85 percent employed in related jobs, and between 70 and 75 percent employed 
in unrelated jobs. Tiiis finding is consistent with findings from nation wide 
studies reported by Marshall ( Manpower, Nov. 1972, p. 8). 

Conclusions and Implications 

A Standards Approach to Program Evaluation. Data from teachers of 
home economics occupation courses and programs were elicited for the purpose 
of assessing whether some standards, drawn from funding legislation and other 
sources, were met. These include the qualifications of professional staff, 
student-teacher ratio, instructional period, use of local employment data, 
provision for work experience, students with special needs, loss of students, 
and use of advisory committees. 



Instruction in home economics occupation courses is proxided by 
teachers who, with a few exceptions, have earned Bachelors degrees and 
approximately 30 percent have Masters degrees. On the criterion of formal 
education, the teachers are qualified to plan and implement curriculum. Be- 
tween 40 and 85 percent report work experience other than teaching. 

The student-teacher ratio at approximately 14 to 1 is quite favorable 
to effective instruction, as is the slight shift to increased instruction time. 

The fact tliat an increased proportion of teachers reported fairly recent 
surveys of local employment opporhmities may indicate that greater emphasis 
is being given to the need to plan and to implement programs in relation to 
manpower demand. 

All courses and programs provided either simulated (approximately 
60 percent) or actual work experience (approximately 40 percent). 

The courses evaluated were serving large proportions of students from 
low income families and disadvantaged families. Students were predominantly 
of average and below ability levels. Although, the proportion of minority 
students was comparatively low at 16 percent in 197 1> it did increase in 1972 
to 24 percent. 

Marshall (Manpower , November, 1972, p. 5) reports that the 20 percent 
minority enrollment figure in the nation, in 1969, is believed to be higher than 
the actual percent enrolled. Within this perspective, the programs in New 
Jersey participating in the study compare quite favorably. Marshall also reports 
that for secondary, post-secondary, and adult vocational programs the ratio of 
completions to enrollments ranges between 33 and 40 percent (p. 8). The 9 per- 
cent loss of students reported for the 23 programs participating in 1972 provides 
one basis for concluding that the holding power of programs is quite good. The 
presence of other intervening variables is acknowledged. 

The discrepancy between the importance placed on ad\isor\' councils by 
legislation, and the formation and use made of advisory councils by local educa- 
tion agencies points to the possibility that some programs are not complying 
with the law. A careful study of impediments to effective «se of advisory 
councils might be in order. 

Student Outcomes: A Basis of Program Evaluation. Instruction did 
result in an increase in knowledge of course content for students in the areas 
of clothing and co-operative education. Student pre -test scores indicated a 
reasonable level. of knowledge of course content in all four areas • This maj^ have 
resulted from instruction in previous home economics courses and thus course 
instruction in the occupation course reinforced prior learning. This maj'^ partiallj^ 
explain the no change for food service and child care whereas, some appreciable 
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gains were made by students in clothing related and co-operative education 
courses. The limitation of testing course content by paper and pencil testing 
may contribute to the limited evidence of gains particularly in view of the 
ability levels of students and the liklihood of a) limited reading skills; and 
b) negative pre-conditioning to paper-pencil tests, through years of limited 
success in school. 

In the area of self-evaluation in work which is more specifically and 
directly related to the objectives of home economics occupation courses, than 
some of the other attitudes measured, the significant shifts which occurred in 
item responses attest to some degree of success in achieving course objectives. 
Experiences in the course may have been instrumental in some of the significant 
shifts in perceptions of ability, problems in school, interest in occupational 
training, and attitudes toward job which were consistent with course objectives. 

The stability of results on measures of other variables such as general 
level of self-confidence, attitude towards home economics, attitudes toward 
work, and attitudes toward working with others, from several vantage points 
is neither sui'prising nor an indication of failure of instruction. First, these 
variables are generalized attitudes formed through many years of varied ex- 
periences. Findings from social-psychological research indicate that these 
types of attitudes change slowly. Furthermore, the attitudes of students as 
evidenced on the pre-test mean scores were not extreme and thus it is ques- 
tionable as to what and how much change in scores would constitute a positive 
outcome from instruction. A similar perspective helps explain the fairly 
minimal changes in the behavior rated by teachers. These behaviors fall 
within the categozy of habitual modes of interacting with others acquired from 
the pre-school years witliin the family, through the elementary and junior high 
years. Once again, mean scoi^es of students were close to tlie theoretical 
scale means at the time of the pre-test. 

Follow-up data on graduates must be viewed conservatively due to the 
limited number of programs from which data were available. Marshall (Man- 
power , November, 1972, p. 8) reports that nationally, more than l/2 of grad- 
uates from secondary vocational programs go for further schooling. For these 
programs between 30 and 50 percent did not seek employment; further schooling, 
military and marriage were partial explanations. 

Of all high school vocational graduates, according to Mai'shall (1972, 
p. 6) 1/4 enter jobs in their field of training. Of all students responding for 
these programs, in 1971 and 1972, between 30 and 38 percent were employed 
in jobs for which the course prepared them. Approximately 10 percent of the 
total were in related jobs. Thus making the total instructional accountability 
approximately 40 to 48 percent. 

In brief, the record, from available data, indicates that these programs 
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have achieved a relative degree of success in fulfilling program objectives. 



Systematizing Local Education Agency Evaluations: A Basis for State- 
Wide Program Evaluation, Assessment and Recommendations. The body of 
this report provides the best empirical evidence relative to the strengUi and 
w^eakness of using data from local education agencies as a basis for assessing 
program outcomes within the state. Specifically, it was possible to systema- 
tize LEA evaluations by providing uniform data collection tools and specify- 
ing procedures. This uniformity in tools and procedures made it feasible to 
combine data from local programs, to group data in relevant categories and 
to analyze it. The results provide a more comprehensive description and 
cross comparisons between ability categories, occupational areas, and pre- 
testing and post-testing of a larger number of students in the state than has 
been available before. 

According to observations of the state department home economics 
staff, teachers of occupation courses became much more sensitive to and 
knowledgable about evaluation. 

Furthermore, there is every reason to have confidence in the integrity 
with which these teachers cooperated in providing botJi pre-test and post-test 
data. 

In several areas there are some apparent weaknesses. One area is 
that of measurement. Paper and pencil tests have some definite limitations 
as attitude and information measures and particularly so with students who 
are not academically oriented. Similarly observation scales have other 
limitations, and these were, in all liklihood, in operation with teacher ratings 
of student behaviors. One teacher specifically explained that she had not 
returned post-test materials because her class, of predominantly minority 
students from disadvantaged low-income families, found reading the tests 
too difficult and thus avoided the task by random marking of responses. 

A second weakness was in the low proportion of returns of post-test 
materials. Follow-up telephone calls were made to increase responses. 
Some teachers reported atypical circumstances which had developed in their 
positions making the post-testing difficult or the scoring and returning of 
materials impossible. There was a considerable time demand placed on the 
teacher. This, however, must be analyzed in terms of whether the time spent 
on materials for the project reduced some time which was needed previously 
for evaluative processes which she employed. Although some consideration was 
given to developing scoring procedures to be done by project staff, this was not 
implemented, because it thwarted achieving the purpose of immediate feedback 
to the teacher as input to her instruction during the course and at its conclusion. 
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The proportion of returns reduced the extent to wliich findings can be general- 
ized to other programs in the state. 

In conclusion, the following recommendations are presented for con- 
sideration. 

First, at regular intervals, e. g. every three or four years, a repre- 
sentative sample of occupation programs should be drawn. Teachers of these 
programs should be asked to participate aiid every incentive to participate 
throughout a pre-test and post-test be provided. 

Secondly, the procedures for systematizing the LEA evaluation should 
be formulated to restrict tlie variables to be measured to those most directly 
linked to program objectives. Teachers should be involved in analyzing pre- 
sent course content to differentiate new concepts to be presented in the instruc- 
tion, from concepts presented in previous instruction, whiph'^are to be reinforced. 
Such a procedure should improve the balance and direction of effort expended in 
instruction and lead to the development of or refinement of content measures 
to be used in evaluation. 

Thirdly, evaluation procedures and materials used should be made 
available to teachers of all programs in the state for tlieir use in inter- 
vening years. The Vocational Division might advisedly ask that these data 
for each local program be included in the final report made by each LEA. 
The findings from the analysis of data from the representative random sample 
would provide a type of normative basis for assessing the relative effectiveness 
of a specific program. 
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